Papers with word segmentation
A Graph-based Model for Joint Chinese Word Segmentation and Dependency Parsing (2020.tacl-1)
Copied to clipboard
| Challenge: | Chinese word segmentation and dependency parsing suffer from error propagation . a graph-based model can integrate both tasks, but it suffers from performance limitations . |
| Approach: | They propose a graph-based model to integrate Chinese word segmentation and dependency parsing . their model achieves better performance than previous joint models . |
| Outcome: | The proposed model achieves better performance than previous joint models and state-of-the-art results in both Chinese word segmentation and dependency parsing. |
A Probabilistic Toolkit for Multi-grained Word Segmentation in Chinese (2025.coling-demos)
Copied to clipboard
| Challenge: | Existing tools for word segmentation are based on different linguistic theories or target different scenarios. |
| Approach: | They propose a probabilistic toolkit for multi-grained word segmentation in Chinese . they adopt semi-Markov CRF for single-grain word segmenting (SWS) . |
| Outcome: | The proposed approach can produce marginal probabilities of words during inference and significantly improve performance in the cross-domain scenario. |
SanskritShala: A Neural Sanskrit NLP Toolkit with Web-Based Interface for Pedagogical and Annotation Purposes (2023.acl-demo)
Copied to clipboard
| Challenge: | SanskritShala is a neural-based Sanskrit NLP toolkit that is available as a web-based application . |
| Approach: | They propose a neural Sanskrit NLP toolkit that facilitates linguistic analyses for word segmentation, morphological tagging, dependency parsing, and compound type identification. |
| Outcome: | The proposed toolkit reports state-of-the-art performance on benchmark datasets . it is built with easy-to-use interactive data annotation features . |
VnCoreNLP: A Vietnamese Natural Language Processing Toolkit (N18-5)
Copied to clipboard
| Challenge: | Using word segmenters and POS taggers, Vietnamese NLP pipelines are no longer considered SOTA models for Vietnamese. |
| Approach: | They propose a Java NLP annotation pipeline for Vietnamese that provides rich linguistic annotations. |
| Outcome: | The proposed toolkit provides rich linguistic annotations to facilitate research work on Vietnamese NLP. |
A Robustly Optimized BMRC for Aspect Sentiment Triplet Extraction (2022.naacl-main)
Copied to clipboard
| Challenge: | Aspect sentiment triplet extraction (ASTE) is a challenging subtask in aspect-based sentiment analysis. |
| Approach: | They propose a bidirectional machine reading comprehension method to extract triplets of aspects, opinions and sentiments with complex correspondence from the context. |
| Outcome: | The proposed method achieves state-of-the-art on multiple benchmark datasets. |
Our Neural Machine Translation Systems for WAT 2019 (D19-52)
Copied to clipboard
| Challenge: | In the last five years, statistical machine translation is gradually fading out in favor of neural machine translation. |
| Approach: | They describe a novel Neural Machine Translation (NMT) system for the WAT 2019 translation tasks they focus on. |
| Outcome: | The proposed system improves translation accuracy while replacing absolute position representations with relative positions. |
Improving Chinese Story Generation via Awareness of Syntactic Dependencies and Semantics (2022.aacl-short)
Copied to clipboard
| Challenge: | Current neural models for Chinese story generation struggle to generate high-quality long text narratives due to ambiguity in syntactically parsing the Chinese language. |
| Approach: | They propose a framework that enhances the feature capturing mechanism by informing the generation model of dependencies between words and additionally augmenting the semantic representation learning through synonym denoising training. |
| Outcome: | The proposed framework outperforms the state-of-the-art Chinese generation models on all evaluation metrics, showing that it enhances dependency and semantic representation learning. |
AraDIC: Arabic Document Classification Using Image-Based Character Embeddings and Class-Balanced Loss (2020.acl-srw)
Copied to clipboard
| Challenge: | a novel image-based character embedding framework is used for text classification in Arabic . classical methods require morphological analysis, word segmentation, and hand-crafted feature engineering. |
| Approach: | They propose a novel end-to-end Arabic document classification framework, Arabic document image-based classifier, inspired by image-basic character embeddings. |
| Outcome: | The proposed framework improves on modern standard Arabic, colloquial Arabic, and Classical Arabic. |
Unsupervised Boundary-Aware Language Model Pretraining for Chinese Sequence Labeling (2022.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that Boundary-Aware BERT can improve Chinese sequence labeling tasks. |
| Approach: | They propose to encode boundary information directly into pre-trained language models . they propose to use unsupervised boundary information instead of supervised boundary info . |
| Outcome: | The proposed architecture improves Chinese sequence labeling tasks on ten benchmarks. |
Probabilistic Bilingual Subword Segmentation with Latent Subword Alignment (2026.eacl-srw)
Copied to clipboard
| Challenge: | Existing methods do not consider parallel relationships, preventing translation model training. |
| Approach: | They propose a method for learning subword correspondences in parallel sentence pairs using the EM algorithm. |
| Outcome: | The proposed method improves translation accuracy for many tasks. |
Post-ASR Correction in Hindi: Comparing Language Models and Large Language Models in Low-Resource Scenarios (2026.eacl-short)
Copied to clipboard
| Challenge: | Automatic Speech Recognition (ASR) systems for low-resource languages produce erroneous transcripts due to limited annotated data and linguistic complexity. |
| Approach: | They compare language models and large language models for post-ASR correction in Hindi . they observe a scaling trend under zero-shot ICL where mid-sized LLMs degrade performance before marginal recovery at extreme scales. |
| Outcome: | The proposed model outperforms larger models in both fine-tuning and in-context learning settings. |
KWJA: A Unified Japanese Analyzer Based on Foundation Models (2023.acl-demo)
Copied to clipboard
Nobuhiro Ueda, Kazumasa Omura, Takashi Kodama, Hirokazu Kiyomaru, Yugo Murawaki, Daisuke Kawahara, Sadao Kurohashi
| Challenge: | KWJA supports a wide range of tasks including typo correction, word segmentation, word normalization, named entity recognition, dependency parsing, PAS analysis, bridging reference resolution, coreference resolution, and discourse relation analysis. |
| Approach: | They propose to build a Japanese text analyzer based on foundation models that performs a wide range of tasks. |
| Outcome: | The proposed model performs better in a multi-task manner than other analyzers with specialized models. |
WSpeller: Robust Word Segmentation for Enhancing Chinese Spelling Check (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Chinese spelling check (CSC) detects and corrects spelling errors in Chinese texts. |
| Approach: | They propose a Chinese spelling check model that takes into account word segmentation and a module that can assist the correction module by predicting correct word segmentations from sentences containing spelling errors. |
| Outcome: | The proposed model outperforms baselines on SIGHAN13, SIGHEN14, and SIGHAN15 and maintains equal performance on SSGHAN14. |
Handling Cross- and Out-of-Domain Samples in Thai Word Segmentation (2021.findings-acl)
Copied to clipboard
Peerat Limkonchotiwat, Wannaphong Phatthiyaphaibun, Raheem Sarwar, Ekapol Chuangsuwanich, Sarana Nutanong
| Challenge: | Word segmentation is domain-dependent, which can be a challenge in low-resource languages like Thai and Urdu . a framework to handle out-of-domain inputs is proposed to improve word segmentation . |
| Approach: | They propose a domaingeneric domain adaptation framework and data augmentation technique to combat low-resource problems. |
| Outcome: | The proposed model outperforms the state-of-the-art Thai word segmentation method in out-of domain scenarios. |
PrahokBART: A Pre-trained Sequence-to-Sequence Model for Khmer Natural Language Generation (2025.coling-main)
Copied to clipboard
| Challenge: | Pre-trained sequence-to-sequence models are typically pretrained on extensive raw text corpora and fine-tuned on task-specific data. |
| Approach: | They introduce a pre-trained sequence-to-sequence model trained from scratch for Khmer using carefully curated Khmer and English corpora. |
| Outcome: | The proposed model outperforms existing models on three generative tasks and is data-efficient and effective in enhancing performance across various natural language generation tasks. |
ERAS: Evaluating the Robustness of Chinese NLP Models to Morphological Garden Path Errors (2025.naacl-long)
Copied to clipboard
| Challenge: | In languages without orthographic word boundaries, NLP models perform _word segmentation_ as an explicit preprocessing step or as an implicit step in an end-to-end computation. |
| Approach: | They propose a benchmark that tests a model’s vulnerability to morphological garden path errors by comparing its behavior on local segmentation ambiguities. |
| Outcome: | The proposed benchmark compares a model's behavior on sentences with and without local segmentation ambiguities and shows that models with character-level tokenization make implicit garden path errors even without an explicit word segmentation step. |
ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information (2021.acl-long)
Copied to clipboard
| Challenge: | ChineseBERT model incorporates glyph and pinyin information of Chinese characters into pretraining . proposed model achieves new performance boost over baseline models with fewer training steps . |
| Approach: | They propose a ChineseBERT model that incorporates glyph and pinyin information into pretraining . the glyph embedding is obtained based on different fonts of a character, and the pinyink embeddment characterizes the pronunciation of Chinese characters. |
| Outcome: | The proposed model achieves new performance boosts over baseline models with fewer training steps. |
A Pragmatic Approach for Classical Chinese Word Segmentation (L18-1)
Copied to clipboard
| Challenge: | Classical Chinese word segmentation is largely neglected due to its obsoleteness . a new approach to segmentation using a marked-up corpus is needed . |
| Approach: | They propose a pragmatic approach to deal with Classical Chinese word segmentation without any marked-up corpus. |
| Outcome: | The proposed method makes the CCWS without any marked-up corpus more accurate compared with collocation-based methods. |
Fluid Annotation: A Granularity-aware Annotation Tool for Chinese Word Fluidity (L18-1)
Copied to clipboard
| Challenge: | Using word segmentation, we propose a wordhood annotation framework for Chinese language . word segmentations have been used for years in preprocessing NLP tasks for languages without explicit word delimiter. |
| Approach: | They propose a word-granularity-aware annotation framework for Chinese language . they argue that word segmentation is fluid in nature and that it rearranges the boundary of word segmentations and linguistic annotation. |
| Outcome: | The proposed framework rearranges the boundary between word segmentation and linguistic annotation and supports flexible annotation tasks for various linguistic and affective phenomena. |
Urdu Word Segmentation using Conditional Random Fields (CRFs) (C18-1)
Copied to clipboard
| Challenge: | Urdu is amongst languages for which word segmentation is a complex task as it exhibits space omission and space insertion issues. |
| Approach: | They propose a word segmentation system for Urdu which uses a Conditional Random Field sequence modeler with orthographic, linguistic and morphological features. |
| Outcome: | The proposed system achieves an F1 score for word boundary identification and 0.85 for sub-word boundary identification tasks. |
DASA-Trans-STM: Adaptive Efficient Transformer for Short Text Matching using Data Augmentation and Semantic Awareness (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models have shown impressive versatility across various tasks. |
| Approach: | They propose a novel adaptive Transformer for Chinese short text matching using data augmentation and semantic awareness. |
| Outcome: | The proposed model can deal with word ambiguity in Chinese on four available datasets. |
Free as in Free Word Order: An Energy Based Model for Word Segmentation and Morphological Tagging in Sanskrit (D18-1)
Copied to clipboard
Amrith Krishna, Bishal Santra, Sasi Prasanth Bandaru, Gaurav Sahu, Vishnu Dutt Sharma, Pavankumar Satuluri, Pawan Goyal
| Challenge: | a structured prediction framework is proposed to solve word segmentation and morphological tagging tasks in a free word order language. |
| Approach: | They propose a structured prediction framework that jointly solves word segmentation and morphological tagging tasks in Sanskrit. |
| Outcome: | The proposed model outperforms the state of the art with an F-Score of 96.92 (percentage improvement of 7.06%) while using less than one tenth of the task-specific training data. |
Incorporating Word Attention into Character-Based Word Segmentation (N19-1)
Copied to clipboard
Shohei Higashiyama, Masao Utiyama, Eiichiro Sumita, Masao Ideuchi, Yoshiaki Oida, Yohei Sakamoto, Isaac Okada
| Challenge: | Word segmentation models are used to minimize the effort in feature engineering. |
| Approach: | They propose a character-based model that learns the importance of multiple candidate words for a corresponding character on the basis of an attention mechanism and makes use of it for segmentation decisions. |
| Outcome: | The proposed model outperforms the state-of-the-art models on Japanese and Chinese benchmark datasets. |
Shrinking Japanese Morphological Analyzers With Neural Networks and Semi-supervised Learning (N19-1)
Copied to clipboard
| Challenge: | Modern neural morphological analyzers consume gigabytes of memory. |
| Approach: | They propose a method which uses unigram character embeddings to train a model on labels produced by a state-of-the-art analyzer. |
| Outcome: | The proposed model outperforms dictionary-based methods in Japanese and Chinese . it uses less than 15 megabytes of space and is much smaller than the dictionary- based one . |
A Fine-Grained Domain Adaption Model for Joint Word Segmentation and POS Tagging (2021.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that joint models of word segmentation and POS tagging can lead to better performance because they are closely related. |
| Approach: | They propose a domain adaption method for Chinese word segmentation and POS tagging that uses a simple metric to model the gaps between target and target domains. |
| Outcome: | The proposed method can gain significant performance improvements over baselines on a benchmark dataset. |
How to Split: the Effect of Word Segmentation on Gender Bias in Speech Translation (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for subword splitting penalize the representation of feminine linguistic markings. |
| Approach: | They propose a method that preserves subword splitting while leveraging character-based segmentation to properly translate gender. |
| Outcome: | The proposed approach preserves BPE overall translation quality while leveraging the higher ability of character-based segmentation to properly translate gender. |
Is Word Segmentation Necessary for Deep Learning of Chinese Representations? (P19-1)
Copied to clipboard
| Challenge: | Using word-based models, we compare word-oriented models with char-based ones . word-driven models are more vulnerable to data sparsity and the presence of out-of-vocabulary words . |
| Approach: | They benchmark word-based models with char-based model which does not involve word segmentation in four NLP benchmark tasks. |
| Outcome: | The proposed model outperforms char-based models in four NLP benchmark tasks. |
Segmentation-free compositional n-gram embedding (N19-1)
Copied to clipboard
| Challenge: | Existing word embedding models depend on word segmentation, but this method is difficult when corpora written in noisy or unsegmented languages. |
| Approach: | They propose a new method that models words, phrases and sentences seamlessly without word segmentation. |
| Outcome: | The proposed method is very effective for noisy corpora written in unsegmented languages such as Chinese and Japanese. |
A Pilot Study for Chinese SQL Semantic Parsing (D19-1)
Copied to clipboard
| Challenge: | Existing datasets to map natural language text into SQL are limited in their use in question-to-sql mapping. |
| Approach: | They propose to use a Chinese-based semantic parser to map natural language text into SQL. |
| Outcome: | The proposed dataset compares a character-based parser with a word embedding scheme for Chinese . the results show that the parsers are subject to segmentation errors and cross-lingual embedders are useful for text-to-SQL mapping. |
Joint Persian Word Segmentation Correction and Zero-Width Non-Joiner Recognition Using BERT (2020.coling-main)
Copied to clipboard
| Challenge: | a set of rules are set by the regulatory body of the Persian language regarding the use of white space and ZWNJ . only a few people follow these rules in writing formal Persian, let alone the informal language. |
| Approach: | They address problems of word segmentation and zero-width non-joiner recognition in Persian . they use a macro-averaged F1 score of 92.40% on a carefully collected corpus of 500 sentences . |
| Outcome: | The proposed problem is a sequence labeling problem in Persian . it achieves a macro-averaged F1 score of 92.40% on a carefully collected corpus of 500 sentences . |
A Fast and Accurate Vietnamese Word Segmenter (L18-1)
Copied to clipboard
| Challenge: | Experimental results show that our approach outperforms previous state-of-the-art approaches in terms of accuracy and performance speed. |
| Approach: | They propose a method where rules are stored in an exception structure and new rules are only added to correct segmentation errors. |
| Outcome: | The proposed approach outperforms existing methods on Vietnamese treebank benchmarks. |
That Slepen Al the Nyght with Open Ye! Cross-era Sequence Segmentation with Switch-memory (2022.acl-long)
Copied to clipboard
| Challenge: | Grammar, vocabulary, and lexical semantic shifts take place over time, creating diachronic linguistic gap. |
| Approach: | a Chinese cross-era learning framework incorporates era-specific linguistic knowledge into a neural network. |
| Outcome: | a new learning framework for Chinese word segmentation improves on four corpora from different eras. |
Neural Graph Matching Networks for Chinese Short Text Matching (2020.acl-main)
Copied to clipboard
| Challenge: | Chinese word segmentation can be erroneous, ambiguous or inconsistent, causing performance problems. |
| Approach: | They propose a sentence matching framework that uses paired word lattices as input instead of a character sequence. |
| Outcome: | The proposed framework outperforms the state-of-the-art short text matching models on two Chinese datasets. |
Improved Unsupervised Chinese Word Segmentation Using Pre-trained Knowledge and Pseudo-labeling Transfer (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to unsupervised Chinese word segmentation require multiple inferences to perform word segmenting. |
| Approach: | They propose a method that integrates the segmentation signal from an unsupervised language model to a pre-trained BERT classifier under a pseudo-labeling framework. |
| Outcome: | The proposed method achieves state-of-the-art performance on the eight UCWS tasks while significantly reducing training time compared to previous approaches. |
Learning to Discover, Ground and Use Words with Segmental Neural Language Models (P19-1)
Copied to clipboard
| Challenge: | Existing models of word learning do not account for the long-range dependencies manifest in language and that are easily captured by recurrent neural networks. |
| Approach: | They propose a segmental neural language model that unifies word discovery, learning how words fit together to form sentences, and by conditioning the model on visual context, how words’ meanings ground in representations of nonlinguistic modalities. |
| Outcome: | The proposed model learns predictive distributions better than character LSTM models, discovers words competitively with nonparametric Bayesian word segmentation models, and improves on both. |
Lattice Transformer for Speech Translation (P19-1)
Copied to clipboard
| Challenge: | Recent advances in sequence modeling have highlighted the strengths of the transformer architecture. |
| Approach: | They propose a general lattice transformer for speech translation where the input is the output of the automatic speech recognition (ASR) they propose 'controllable' lattica attention mechanism to consume latent representations. |
| Outcome: | The proposed model outperforms baseline and lattice LSTM on the Chinese-English translation task. |
PyCantonese: Cantonese Linguistics and NLP in Python (2022.lrec-1)
Copied to clipboard
| Challenge: | a limited number of Cantonese-specific datasets are available for PyCantones. |
| Approach: | They introduce PyCantonese, an open-source Python library for Cantonesi linguistics and natural language processing. |
| Outcome: | The proposed library is open-source and available for free for all purposes, including commercial ones. |
A Multitask Learning Approach for Diacritic Restoration (2020.acl-main)
Copied to clipboard
| Challenge: | Diacritics are used to specify pronunciations and meanings in many languages like Arabic. |
| Approach: | They propose to use multi-task learning to optimize diacritic restoration with related NLP problems . they use Arabic as a case study since it has sufficient data resources for tasks . |
| Outcome: | The proposed model outperforms baseline models and is comparable to the state-of-the-art models. |
Improving Chinese Named Entity Recognition with Multi-grained Words and Part-of-Speech Tags via Joint Modeling (2024.lrec-main)
Copied to clipboard
| Challenge: | Named entity recognition (CNER) is a fundamental task in natural language processing (NLP). |
| Approach: | They propose a tree parsing approach for jointly modeling Chinese named entity recognition (CNER) with multi-grained word segmentation (MWS) and POS tagging tasks. |
| Outcome: | The proposed approach achieves better or comparable performance with current methods. |
Segment First or Comprehend First? Explore the Limit of Unsupervised Word Segmentation with Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to measure word segmentation only assess the language model's understanding of the overall meaning of sentences, lacking an evaluation of the language models' understanding capabilities at a fine-grained level. |
| Approach: | They propose a framework to explore the limit of unsupervised word segmentation with Large Language Models (LLMs) they employ current mainstream LLMs to perform word segmentations across multiple languages . |
| Outcome: | The proposed method improves on existing methods and combines the advanced pattern recognition capabilities of Aho-Corasick automata with the deep insights of well-pretrained LLMs. |
Multi-Tiered Cantonese Word Segmentation (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing work on word segmentation for Chinese does not have conventional word boundaries as English does. |
| Approach: | They propose a linguistically motivated, multi-tiered word segmentation system for Cantonese . they propose linguisticly motivated, linguistic-motivated system that can cater to different needs . |
| Outcome: | The proposed system can be adapted to Cantonese corpus data. |